Assessing L2 Pragmatic Proficiency using a Live Chat Platform: A Study on 


Compliment Sequences 


Jeremy W. Bachelor 


English Literacy and Modern Languages, Heartland Community College, Normal, IL, USA 


The Version of Record of this manuscript has been published and is available in The Language 
Learning Journal, January 12, 2020, https://www.tandfonline.com/loi/rllj20 


To cite this article: Jeremy W. Bachelor (2020): Assessing L2 pragmatic proficiency using a 
live chat platform: a study on compliment sequences, The Language Learning Journal, DOI: 
10.1080/09571736.2019.1709887 


Assessing L2 Pragmatic Proficiency using a Live Chat Platform: A Study on 


Compliment Sequences 


The objective of this study was to employ a new tool for assessing pragmatic proficiency 
(over performance) in L2 students and to determine if pragmatic lessons on compliment 
sequences had a positive impact on L2 Spanish students’ intercultural competence. To this 
end, students at a Midwestern college engaged in live video chat pre-test conversations in 
Spanish in which a native speaker was instructed to compliment the student as to naturally 
assess the response. The student was not aware that s/he would be complimented; as such, 
this tool may be the most authentic way to assess student pragmatic proficiency within a 
classroom setting. For comparison purposes, a multiple choice test was also administered. 
After the pre-tests, pragmatic interventions on compliment sequences were presented in 
class, followed by a repeat of the pre-tests (now post-tests). Growth between the tests was 
the main focus of the data analysis process. Statistical and anecdotal results indicate that 
the pragmatic lessons greatly increased the students’ ability to recognise appropriate 
compliment sequences in Spanish, but not to produce them. While these findings are not 
ground-breaking, the innovative nature of the assessment method used should encourage 
future researchers to use similar assessment techniques for measuring pragmatic 


proficiency. 
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Introduction 

Language teachers increasingly understand the importance of developing students’ intercultural 
competence by focusing on pragmatic speech acts (Vellenga, 2011). This can be largely 
attributed to the acceptance by many educators that grammar-driven classroom approaches do 
not suffice and that a focus on promoting effective communication among students is vital to 
language acquisition (Bachelor, 2017; Bachelor, 2019). To document this acceptance, the most 
recent ‘NCSSFL-ACTFL Can-Do Statements’ (2017) include an intercultural communication 
tier, which emphasises student behaviour in target language interactions. According to these 
‘Can-Do’ statements (p. 9), students must be able to ‘avoid major social blunders’ and ‘transition 
smoothly from formal to informal styles of behaviour,’ among others, which are main goals of 
pragmatic instruction. 

Since research has demonstrated that pragmatic errors tend to be more significant than 
grammar errors in achieving effective communication (Barros Garcia & Bachelor, 2018; Wolfe, 
Shanmugaraj, & Sipe, 2016), it is crucial that students understand the nuances of the language 
that they are studying, which in turns increases a student’s intercultural competence. While this 
has been acknowledged by many teachers, these same educators are simply not trained in how to 
incorporate these components or lack evidence as to the effectiveness of their teaching strategies 
(Vellenga, 2011). 

Likewise, L2 educators who do include pragmatics in their curriculum are often unsure 
how to measure the pragmatic ability of their students (Cohen, 2008). While Discourse 
Completion Tasks (DCTs) and role plays are often used to measure specific pragmatic elements 
of students, these assessment scenarios are often contrived and only measure pragmatic 


performance rather than pragmatic proficiency. Even real play activities in which students 


‘interact with real-world individuals as part of class activities’ (Cohen, 2008, p. 220) allow for 
the student to prepare for the pragmatic event as s/he would for a role play. As such, teachers are 
further tasked with how to best demonstrate the effectiveness of their pragmatic lessons. 

Compliments and compliment responses are among the most widely used speech acts 
across cultures (Duan, 2011). In fact, Barros Garcia (2011) found in her analysis of language 
corpus regarding face-enhancing politeness in informal Spanish that compliments and their 
responses accounted for more than any other speech act employed by native speakers. However, 
Smith (2009) found that compliments and ‘CR [compliment response] strategies are not 
commonly taught in L2 classrooms’ (p. 115). This highlights a significant problem, as the 
strategies employed when complimenting and responding vary greatly from culture to culture 
(Yoko, 1995). It is essential that L2 students understand how compliments in the target language 
vary from those of their own and how to appropriately respond. As Yousefvand, Yousofi, and 
Abasi (2014) explain, “speakers need to know when and how it is appropriate to give [some]one 
a compliment or respond to the received compliment’ so as not to be considered “ill-cultured 
while communicating in a foreign language’ (p. 182). For example, Fong (1998) found instances 
where the sincerity of American English speakers was questioned due to the compliment 
strategies used. Similarly, Yoko (1995) discovered that accepting a compliment rather than 
rejecting it can be offensive in some countries. Finally, while generally perceived as something 
positive, compliments may be interpreted as expressing envy or even verbal harassment in 
certain cultures (Nkwain, 2011). Thus, lack of knowledge by the student as to how compliments 
and their responses function “can cause serious problems and result in miscommunication’ 


(Yousefvand, Yousofi, & Abasi, 2014, p. 183). So as not to affront an addressee and cause a 


breakdown in intercultural communication, it is important that students familiarize themselves 
with this important speech act. 

For these reasons, the present study sought to analyse the pragmatic competency in 
relation to compliment strategies and responses of novice and intermediate level Spanish 
students by designing pre and post-tests (recordings of authentic exchanges between students and 
native speakers and a paper and pencil test), implementing pragmatic interventions (lessons) on 
compliments/responses, and journaling classroom experiences. Control and experimental groups 
were established to minimise other variables. The results from this analysis will provide teachers 
with the evidence that they need to support their existing pragmatic interventions or to encourage 
them to create comparable lessons for their own circumstances as to promote interculturality in 
the classroom. In contrast to many studies on student use of particular speech acts, the research 
reported here uses data from authentic occurrences of compliments rather than just contrived 


written tests or role plays. 


Literature review 
Pragmatics and speech acts 
For many years, Language teachers focused primarily on increasing students’ grammatical 
competence, and the teaching methods applied supported memorization and repetition (Bachelor, 
2015). However, a paradigm shift occurred in the 1970s and early 1980s, as L2 educators 
discovered that the main objective of language teaching has to do with effective communication 
(Bachelor, 2016; Bachelor & Barnard Bachelor, 2016). 

In studying effective communication, V.G. and Rajan (2012) found that the 


communication skills of non-native students are negatively affected by a lack of pragmatic 


knowledge. Since communication is the main objective of language learning, the study by V.G. 
and Rajan further suggests that pragmatic instruction is necessary to ensure that students are able 
to communicate effectively in the target language and that grammar-centred instruction alone is 
not sufficient in achieving this goal. 

According to Romero Betancourt (2012), pragmatics is an area of research that focuses 
on linguistic acts within specific contexts. In the field of foreign language teaching, pragmatic 
instruction attempts to facilitate a student’s ability to apply socially appropriate language 
strategies within specific contexts (V.G. & Rajan, 2012). Oftentimes, teachers focus on the use 
of speech acts within differing contexts. 

According to the Center for Advanced Research on Language Acquisition (2016) 
[CARLA], a speech act is a statement or utterance that plays a role in communication such as 
compliments, requests or invitations. Successful Language students must be able to employ the 
correct pragmatic strategies in their production of speech acts in the target language, according to 
appropriate social norms in that speech community, which often differ from their own. 

Langer (2011) explains that speech acts can be divided into three parts: meaning 
(referential value), function (implication of speaker), and effect (inference by addressee). 
Likewise, speech acts may be direct or indirect; for example, “close the window’ is likely 
perceived as a direct request, while ‘it’s cold, right?’ might serve as an indirect request. The 
relationship between speakers also affects how language is used. For instance, the social distance 
between the interlocutors (which is conditioned by factors such as age, gender, and 


socioeconomic level) may result in different linguistic options (Garcia, 2004). 


Compliments and compliment responses 

As mentioned, one of the most commonly used speech acts across languages is a 
compliment/compliment response (Duan, 2011). Compliments are ‘expressions of positive 
evaluation that commonly occur in everyday conversational encounters among interlocutors’ and 
typically occur among interlocutors of similar social status (Félix-Brasdefer, 2011, para 1). They 
are often used to express admiration of someone’s work or appearance, to reinforce desired 
conduct, to establish solidarity, or to soften face-threatening acts (CARLA, 2016). Compliments 
may be external or internal; an external compliment encompasses positive comments on tangible 
items like objects, property, or one’s physical aspect or attire, whereas an internal compliment 
refers to one’s personality, skills, or personal characteristics (CARLA, 2016). Torio Sanchez 
(2004) highlights nine functions for compliments: 

(1) To express admiration or approval of someone’s work, appearance or taste, (2) to 
establish, confirm or maintain solidarity between the interlocutors, (3) to substitute 
acknowledgments, congratulations or greetings, (4) to mitigate acts that threaten one’s face, such 
as apologies or criticism, (5) to initiate and maintain conversations, as a conversational strategy, 
(6) to reinforce appropriate behaviour, (7) to express sarcasm, (8) to ask for the complimented 
object, and (9) to flirt. 

In terms of responses to compliments, Yi (2002) explains that a compliment response can 
be verbal (an audible thank you) or nonverbal (a smile at the interlocutor who directed the 
compliment). In many cultures, responding to a compliment is oftentimes more difficult than 
offering one, since the recipient of a compliment has the task of not only responding to the 
compliment but also avoiding self-praise (Pomerantz, 1978). Additionally, Lazaro Ruiz and 


Ramajo Cuesta (2015) found that ‘there is a conflict between the need of being modest and 


trying to avoid disagreement with the interlocutor. Choosing the right answer from the wide 
range of compliment response strategies causes a dilemma for the complimentee’ (p. 94). In 
choosing the right answer, Brown and Levinson (1987) mention three different strategies that are 
often used that respondents employ to address the above concerns: 

(1) Accept the compliment but detract from it or downgrade it (i.e. oh, this old rag?), (2) 
accept the compliment but attribute the praise to a third person (i.e. the recipe is from my mom), 
or (3) to return the compliment (i.e. I love your shoes too). 

However, an elaborate strategy is not always carried out, as Cedar (2006) found instances 
where a simple acceptance of a compliment without detraction or negative elaboration is 
produced by the complimentee (i.e. thank you, I’m glad you like it). Others choose to elaborate 
positively on the response (i.e. I bought it at the mall), say nothing but smile or laugh, or refuse 


to acknowledge the compliment (no indication of having heard the compliment). 


Compliments and compliment responses in American English 
With respect to American English, nearly 80% of compliments realized by middle-class speakers 
follow one of the following three patterns (Félix-Brasdefer, 2011, para 3): 

1). Noun phrase + is/looks + adjective (i.e. ‘your hair looks nice’) 

2). I like/love + noun phrase (i.e. ‘I like your car’) 

3). Pronoun (a) + adjective + noun phrase (i.e. ‘that’s a nice tie’) 

Additionally, Gajaseni (1994) analysed compliment responses of American English 
speakers and discovered that they are more likely than speakers of other languages to accept a 
compliment and to give a lengthy response. A study by Chiang and Pochtraeger (1993) supports 


these findings and also found that American English speakers rarely reject or deny a compliment. 


Additionally, in her study comparing Thai speakers with American English speakers, 
Cedar (2006) discovered that while the Thai speakers often employed a ‘smile but say nothing’ 
strategy or refused to acknowledge the compliment, the American English speakers never 


employed these strategies and often elaborated positively. 


Compliments and compliment responses in Spanish 

In Spanish, compliments are frequently carried out using a limited number of adjectives and 
verbs (Lorenzo-Dus, 2001) and can be categorized by those on abilities or achievements (i.e., 
your Spanish has really improved!) and by those on appearance or possessions (i.e., nice new 
car!) (Campo & Zuluaga, 2000). 

The most frequent adjectives used to compliment in Spanish are ‘bueno/a,’ ‘bonito/a,’ 
‘guapo/a,’ ‘rico/a,’ and ‘inteligente’ (Lorenzo-Dus, 2001). In terms of speech patterns, Spanish- 
speakers will often use qué [how] plus an adjective, as in qué bonito [how pretty], among other 
structures, as outlined in table 1. 

Table 1. 

In terms of compliment response strategies, Smith (2009) analysed recent studies on 
Spanish compliment responses and concluded that ‘it is widely accepted [among researchers] 
that Spanish speakers are more likely [than speakers of English] to utilize strategies like refusing, 
downgrading, and questioning in responding to compliments’ (p. 106). Smith (2009) also carried 
out his own study and found that Spanish speakers question compliments ‘far more often than 
their native English-speaking peers’ and that they ‘returned a compliment in far fewer situations 
than the English speakers’ (p. 116). Some gender differences in relation to compliment response 


strategies were also found. According to Maiz-Arévalo (2010), rejection or mitigation of the 


compliment is frequent in Spanish, especially among female speakers, in order to create rapport 
and create an environment of equality. 

However, Lorenzo-Dus (2001) found instances where Spanish speakers request a 
repetition or expansion of the compliment by using questions (i.e. do you really think so?). 
Ramajo (2012) supports this finding and also discovered that speakers rarely accept a 
compliment with a simple ‘thank you.’ 

Lazaro Ruiz and Ramajo Cuesta (2015, p. 95) observed a great variety of compliment 
response strategies and different combinations between them. Table 2 demonstrates a summary 
of their classification of compliment responses and is comprised of 2 macro-strategies 
(acceptance and mitigation) that contain several sub strategies. 

Table 2. 

Regardless of the strategy used, Cortés Moreno (2003) writes that it is often difficult for 
native speakers to know how to respond to a compliment, let alone L2 students: 

These expressions [compliments] can refer to beauty, a physical feature or intelligence ... 

Often they tend to plant doubt, even to native speakers themselves, because it is not easy 

to know how we should answer if someone tells us we’re attractive, nice, or intelligent, or 

if they tell us we’re wearing elegant clothes. In the majority of cases, one opts to subtract 

or take importance away from the compliment. (as cited by Smith, 2009, p. 115) 

This comes in stark contrast to strategies used by American English speakers who frequently 
accept a compliment and give a lengthy response (Gajaseni, 1994). 

As previously mentioned, these differences should be taught in the Spanish Language 

classroom as to avoid offending the hearer, as pragmatic errors of this nature are perceived as 


more severe than grammar or pronunciation mistakes (Bardovi-Harlig & Dornyei, 1998; 


10 


Schauer, 2006). Intercultural competence should be at the forefront of the language classroom, 
and pragmatic instruction may help teachers arrive there. To this end, lessons were developed for 
Spanish classes on compliments and compliment responses, as outlined in the forthcoming 


sections. 


Methodology 

Population and sample 

According to U.S. News & World Report (2014), School X is a community college with 5,215 
students, 2,241 of whom attend on a full time basis. The ethnicity of the institution is primarily 
White, at approximately 75% of students. Black students encompass 10% of the total population. 
31% of students receive grant or scholarship aid, and 31% are the first in their family to attend 
college. The average student age is 24, of whom 54% are female and 46% are male. Finally, 23% 
of students successfully graduate, and an additional 26% transfer to a four-year school. In terms 
of instructors, the school employs 85 full time teachers and 192 part time teachers, with a 21:1 
student to teacher ratio. School X’s location is metropolitan, located approximately 130-160 
miles from two megacities in the Midwestern section of the United States. Several connecting 
small towns, along with this mid-size city, comprise the student population. 

School X typically offers four sections of Spanish One (Beginner Spanish I) and one 
section of Spanish Three (Intermediate Spanish I) each fall. Generally, the students in these 
courses are college freshmen or sophomores; however, juniors and seniors from other institutions 
occasionally enrol. 

In consultation with the students, school advisors enrolled the participants across sections 


and later made some adjustments to balance class sizes and to meet individual needs. The 
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classroom teacher taught three sections of Spanish One and one section of Spanish Three during 
the fall of 2017. Students ranged from 18 to 38 years of age at the onset of the study, with 23 as 
the average age. To choose the sample, the researcher designated the first section of the day 
(group one) the control group, consisting of 16 students, in which 8 fully participated in all 
aspects of the study. This group had seven females and one male. The second and third sections 
(group two) were the experimental group, comprised of 27 students, in which 15 fully 
participated. This group had nine females and six males. A final group (group three) of seven 
students from Spanish Three (out of nine enrolled in the course), comprised of six females and 
one male, also fully participated. A control group of Spanish Three students was not possible due 
to the small number of participants enrolled and the lack of a second section. The researcher 
designated the experimental and control groups from Spanish One as such before seeing the lists 
of students in each course to reduce any possibility of bias. This study was conducted with 30 
total participants out of 52, and there were a total of 102 Spanish students college-wide during 
the semester in question. 

At the onset of the study, unpaired f¢ tests were employed to determine whether or not the 
control and experimental groups of Spanish One students entered the study on equal footing. 
Data from the MCDCT pre-tests (later explained) between these groups were compared to make 
this determination. There was not a significant difference in the pre test scores from the control 
group (M=30, SD=19) and the experimental group (M=33.33, SD=20.59); t(0.36)=20, p=0.72. As 
such, the students in both groups started the study with no statistically significant difference in 


their pragmatic competency with regard to compliment sequences. 
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Design 
Permission was granted by the Institutional Review Board and consent was obtained from all 
parties involved before the study began. This is a mixed method study and the design is 
experimental. To this end and as previously mentioned, three groups of students participated in 
this study in the fall semester of 2017. Groups one (control) and two (experimental) were both 
enrolled in Spanish One and group three was enrolled in Spanish Three. Spanish One is a typical 
first semester college-level course intended for those with little or no knowledge of the language 
and covers all forms of the present tense through the present progressive. Spanish Three is a third 
semester course intended for students who previously took Spanish One and Two and focuses 
primarily on the subjunctive while perfecting previously covered concepts. Both courses follow 
the communicative language teaching approach and integrate relevant culture and vocabulary. 
Two pre-tests were administered in all three groups at midterm, just before the 
interventions began. As the Spanish One groups had no knowledge of the language at the onset 
of the semester, the pre-tests would have been meaningless if administered earlier. The first pre- 
test consisted of a live 30-minute conversation in Spanish via TalkAbroad, an online video chat 
platform that matches students with a native speaker. During the first TalkAbroad conversation, 
students were instructed to simply get to know the native speaker using only Spanish. Through 
the online platform, the teacher is able to provide the native speaker with secret instructions as to 
guide the topic. In this case, the speaker was instructed to give the student a compliment 
regarding his or her language ability at some point throughout the conversation as to put the 
student on the spot and assess his or her natural ability to respond to a compliment. This was the 
only provided instruction. The student was not aware in advance that he or she was going to be 


complimented. The second pre-test was a multiple choice discourse completion test (MCDCT) 
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that assessed both compliments and their responses (see Appendix A). It consisted of ten 
questions, five compliments and five compliment responses; of those, five were formal 
situations, and five were informal situations. For each question, the student was presented with a 
scenario and had to choose the most appropriate response. MCDCTs have been used with 
success in pragmatic research with students in a number of experimental studies (Bachelor, 2016; 
Bachelor, 2015; Liu, 2007; Jianda, 2006; Roever, 2006; Roever, 2005). As such, the researcher 
felt that it was a reliable tool for assessing student knowledge and pragmatic recognition. The 
‘correct’ answers on the MCDCT were real answers that were provided by ten native Spanish 
speakers who were given a DCT of the same scenarios before the study began. 

After the pre-tests, three interventions occurred, consisting of three pragmatic lessons on 
compliments and compliment responses in Spanish. These interventions did not take place in the 
control group and constituted the only difference between groups one and two. Group three 
underwent the same pragmatic interventions; however, course content on non-lesson days was at 
the intermediate level. 

The first lesson (see Appendix B) explicitly explained the basics of compliments and 
their responses in Spanish with examples, followed by comprehension exercises to give students 
the opportunity to practice the forms and ideas they studied in the lesson. The final questions 
asked students to think critically about why a certain compliment was given over another one. 

The second lesson (see Appendix C) consisted of three short clips from Spanish-speaking 
shows from Spain and Latin America in which compliments and responses are included. A 
transcription of the dialogues was provided to the students. After watching the clips a couple of 
times each, students were asked to identify the compliments and their responses, the relationship 


between the speakers, and the purpose of the compliments. 
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The third and final lesson consisted of a modified version of the Center for Advanced 
Research on Language Acquisition (CARLA)’s lesson on compliment sequences in Spanish 
(2006). The lesson was condensed and the video clips were transcribed rather than played in 
class as to save time. Students completed the shortened lesson in pairs. The lesson included an 
explicit explanation of compliments and their responses, examples in the form of dialogues, and 
follow up questions regarding the purpose of the compliment and certain social factors that might 
influence the compliment and its response. 

After the intervention concluded, the pre-tests were administered again as post-tests. 
Additionally, the course instructor was asked to keep a journal during the pragmatic lessons. He 
was asked to write down any and all observations regarding student attendance on lesson days, 
student engagement, peer conversations, and answers provided on the lesson worksheets. As the 
lesson activities were not collected or graded by the teacher or by the researcher, insight from the 
journal is essential to analyse student comprehension during the lessons. 

Approximately two months passed between the pre and post-tests, with the pragmatic 


interventions occurring every other week after the pre-test and before the post-test. 


Research questions and data analysis procedures 
The study was designed in an attempt to answer the following research questions: 

1. What effect do pragmatic lessons regarding compliments and compliment responses 
have on L2 Spanish students’ ability to compliment/respond appropriately (according 
to accepted pragmatic norms)? 

2. Do pragmatic lessons regarding compliments and compliment responses in Spanish 


have a more positive impact on beginning or on intermediate level L2 students? 
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3. Are L2 students of Spanish able to transfer classroom knowledge regarding 
compliment responses to the real world? 
In order to respond to these questions, certain analytical procedures were put into place. 
Before answering the research questions, the data from the live TalkAbroad conversations were 
analysed using criteria based on investigations by Smith (2009), Maiz-Arévalo (2010), Lorenzo- 
Dus (2001), and Ramajo (2012) so as to determine what was considered a pragmatically 
appropriate compliment response in standard Spanish. Drawing on insights from these studies, 


the researcher looked for the following strategies when responding to a compliment: 


. Student rejects the 
compliment 

: Student downgrades the 
compliment 

° Student attributes the 


compliment to someone else 
s Student questions or doubts 
the compliment 
° Student asks for an expansion 
of the compliment 


Other characteristics, such as pronunciation or grammar mistakes, were not factors in the 
data analysis process. The researcher listened to each 30-minute conversation from both the pre 
and post speaking tests, transcribed the compliment and compliment response sequences, and 
marked them as either ‘adheres’ or ‘does not adhere’ to pragmatic norms based on the previously 
mentioned criteria. Answers were then given | point for an answer that ‘adhered,’ and 0 points 
for an answer that did ‘not adhere,’ as to assign numerical value to the data. 

The MCDCT (Appendix A) contained 10 questions; students received 0-10 points based 


on the answers selected on both the pre and post-tests. 
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To answer the research questions, paired ¢ tests were used to determine if statistically 
significant growth occurred between the pre and post-tests (naturalistic TalkAbroad 
conversations and the MCDCTs) within all three groups. Additionally, unpaired t tests were 
employed along with effect sizes between groups. 

Furthermore, the teacher journal was analysed to look for any patterns that would indicate 
student comprehension of the lesson material. Finally, the researcher is able to qualitatively 


discern from the recordings why some answers were marked ‘not adhere.’ 


Results 

As mentioned in the prior section, paired f test calculations were used to determine the impact 
the compliment and compliment response lessons had on each group. The MCDCT results were 
processed first, and the paired f test demonstrates that there was an insignificant statistical 
difference within the control group (group one) of Spanish One students between the pre (M=30, 
SD=19) and post-tests (M=34.29, SD=19.02), 1(6)=1.4412, p=0.1996. 

However, in the experimental group (group two) of Spanish One students, the paired rf 
test demonstrates that an extremely statistically significant difference occurred between the pre 
(M=33.33, SD=20.59) and post-tests (M=63.33, SD=17.18), t114)=6.4099, p=0.0001. 

As done at the onset of the study (see Population and sample), an unpaired ¢ test was 
calculated yet again at the end of the study to see if the statistically insignificant difference 
between the Spanish One control and experimental groups became statistically significant by the 
end of the study. The unpaired f test found a very statistically significant difference between the 


control group post test scores (M=34.29, SD=19.02) and experimental group post test scores 
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(M=63.33, SD=17.18); t(20)=3.57, p=0.0019. Similarly, a Cohen’s d effect size resulted in 
d=(63.33-34.29)/18.12=1.6. Such effect size is considered to be a ‘very large effect.’ 

Finally, paired ¢ test results from the MCDCT for Spanish Three students (group three) 
demonstrate that a statistically significant difference occurred between the pre (M=30, 
SD=12.91) and post-tests (M=70, SD=25.17), t(6)=4.3205, p=0.005. 

As for the naturalistic TalkAbroad conversations, paired t tests based on the numerical 
value assigned (see Research questions and data analysis procedures) were also employed in all 
three groups. The paired f test demonstrates that there was an insignificant statistical difference 
within the control group (group one) of Spanish One students between the pre (M=20, 
SD=44.72) and post-tests (M=0, SD=0), t(4)=1.0000, p=0.3739. 

The excerpt below (1) provides an example of an inappropriate response from a student 
from group 1. Before the compliment, the student expresses her difficulty with college in 
general. The native speaker asks if Spanish is a difficult course for her, and she says yes. 

(1) Native speaker (NS) and Student (S) 

1. NS: Pero tu [name of student], hablas muy bien el espanol [but you, you speak Spanish 
really well] 

2. S: ;Gracias! [Thanks!] 

3. NS: En un nivel normal, normal cuando estas aprendiendo espanol, tu hablas muy bien. 
Solo es practica ... Tu [name of student], hablas muy bien [For a normal level, a normal 
level for when you’re learning Spanish, you speak Spanish really well. It’s just practice. 
You speak very well] 


4. S: Si [Yes] 
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Within the experimental Spanish One group (group 2), there was not a statistically 
significant difference between the pre (M=20, SD=41.40) and post-tests (M=20, SD=41.40), 
t(14)=0.0000, p=1.0000. 

The excerpt below (2) provides an example of an appropriate response from a student 
from group 2. The student began by fishing for a compliment and declaring that he is horrible at 
Spanish and also not very smart. 

(2) Native speaker (NS) and Student (S) 

1. NS: Eres muy listo, eres muy listo ... eres muy bueno [You're very smart, you’re very 
smart ... you’re very good] 

2. S: Ah, no ... no soy bueno en espanol [Ah, no, I’m not good at Spanish] 

3. NS: Pero hablas bien ... no eres muy malo. Eres muy bueno, hablas muy bien [But you 
speak well, you aren’t very bad. You’re very good, you speak very well] 

4. S: Gracias [Thanks] 

Finally, within the experimental Spanish Three group (group 3), there was not a 
statistically significant difference between the pre (M=14.29, SD=37.80) and post-tests 
(M=14.29, SD=37.80), t(6)=0.0000, p=1.0000. 

The excerpt below (3) provides an example of an appropriate response from a student 
from group 3. Just before the compliment, the native speaker helps the student with the 
pronunciation of a word. 

(3) Native speaker (NS) and Student (S) 
1. NS: Es muy bien tu pronunciacion. Hablas muy ... hablas muy bien el espanol [Your 


pronunciation is very good. You speak, you speak Spanish very well] 
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2. S: Ah... mas o menos... Estudié espanol para cuatro anos en secundario pero en la 
universidad es muy dificil. Y mis profesoras diferentes cada ano y la transicién es muy 
dificil para mi [Ah, more or less. I studied Spanish for four years in high school but it’s 
really hard in college. And my teachers are different every year and the transition is really 
hard for me] 

3. NS: Pero lo hablas muy bien [But you speak it really well] 

4. S: Gracias [Thanks] 

Table 3 exemplifies the response types provided by students during their post- 
intervention TalkAbroad conversations. As previously demonstrated by the insignificant 
statistical differences, students mostly reverted to the Anglo-Saxon ‘thank you’ (‘gracias’) or 
chose not to reply rather than applying some of the strategies they correctly identified in the 
MCDCT assessment. 

Table 3. 

With regard to the teacher journal, common themes include that the students in the 
experimental groups generally answered questions correctly during the lessons and had a broader 
understanding than before of the motive behind a compliment. More notably, the teacher 
observed on non-lesson days that several students actually used correct compliment sequences 
during in-class role play scenarios. On one instance, a student from group two joked in English 
with his partner and said, ‘Like in that lesson ... ve gotta downplay it!’ 

In sum, statistical data report that students in the experimental groups experienced a 
statistical increase between the pre and post-tests on the MCDCT while the control group did 


not, and that no group experienced a statistical increase between pre and post TalkAbroad 
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conversations. However, anecdotal evidence suggests a greater understanding of the compliment 


sequence among the experimental groups. 


Discussion, implications, and limitations 

In response to research question 1, the results demonstrate that pragmatic lessons regarding the 
compliment sequence in Spanish have a significant impact on a student’s ability to recognise an 
appropriate answer. Within the experimental group from Spanish One, the statistical difference 
between the pre and post-tests was extremely significant on the MCDCT, suggesting that the 
lessons had a very positive impact. This was also observed in the experimental group from 
Spanish Three regarding their ability to recognise appropriate answers on the MCDCT. The 
control group, however, experienced no significant increase, thus suggesting that other variables 
did not play a role in the increase among the students from the experimental group. This is an 
important finding, as the ‘NCSSFL-ACTFL Can-Do Statements’ (ACTFL, 2017, p. 3) suggest 
that students at this level should be able to ‘recognise some obviously inappropriate behaviours 
in familiar everyday situations,’ which is what the experimental group of students were able to 
do on the MCDCT. 

Nevertheless, data from the authentic TalkAbroad conversations suggest that, while 
students may have the ability to discern a ‘right’ response from a ‘wrong’ response, they are not 
able to apply that knowledge to real world situations, as demonstrated by the insignificant 
statistical differences between the pre and post conversations in the experimental groups. As 
demonstrated in Table 3, students’ natural reaction was to revert back to Anglo-Saxon norms 
when responding to a compliment. These data suggest that the preliminary answer to question 3, 


which sought to discover the transferability of classroom pragmatic knowledge to the real world, 
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is ‘no,’ students from the study did not apply their knowledge from the lessons to their authentic 
TalkAbroad conversations. This is not entirely surprising considering the nature of the 
interventions. The lessons did not require students to frequently verbalize a compliment or a 
compliment response in spoken conversations with others; rather, students were asked to identify 
compliments and discuss the intentions and motivations. It is also not the expectation at the 
novice level that students be able to produce appropriate behaviours, but rather, as previously 
mentioned, to simply ‘recognise’ these behaviours as appropriate or not (ACTFL, 2017). 
Regardless, the TalkAbroad platform proved an effective tool to assess pragmatic proficiency 
with regard to compliment responses and demonstrates that more is needed for students to 
transfer classroom knowledge to their interlanguage. 

Additional interventions in future semesters that include compliment sequence role plays 
may better address this concern as students grow in proficiency. However, the teacher journal 
was able to identify situations in which students from the experimental groups did apply 
information from the lessons into classroom role plays; the teacher also noted that students 
seemed nervous during the live chats, which may have had an impact on their ability to apply 
their classroom knowledge to the ‘real world.’ Additionally, there were a number of 
conversations in all groups in which the student simply did not reply or acknowledge the 
compliment, perhaps due to not understanding it or perhaps due to a decision not to. The latter 
would be considered a pragmatic ‘error;’ however, the former would not. Regardless, these 
instances impacted the outcome of the data analysis process. In conclusion, to answer questions | 
and 3, the pragmatic lessons had a positive impact on students’ ability to recognise correct 
pragmatic responses, but they did not have an impact on student oral production in real-life 


circumstances. 
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In terms of research question 2 as to whether the lessons helped beginning or 
intermediate Spanish students more, both groups two and three experienced a statistically 
significant increase; however, the increase in the Spanish One group was ‘extremely significant,’ 
while the increase in the Spanish Three group was only significant. Thus, lessons may have a 
different impact on students at different levels. Given that this research found that the strongest 
impact was on beginning students, it seems plausible to suggest that introducing lessons on target 
language pragmatics from the earliest stages of acquisition may result in the greatest gains over 
time. 

A significant takeaway from this study is the way in which the data was collected. 
Though the idea of pragmatic lessons having a positive impact on L2 students is not in and of 
itself innovative, the data collection process here is. While nothing compares to the naturally and 
secretly recorded conversations done abroad by Shively (2011), the TalkAbroad data collection 
method provides a similar naturalistic environment that connects the student to the target culture 
without travel. One major innovation here is that the students in this present study were not 
aware that they would be complimented. As such, they were put entirely on the spot and had no 
time to prepare a response. This is a way for educators and researchers to measure pragmatic 
proficiency over pragmatic performance. The researcher hopes to inspire future investigations to 
use TalkAbroad and other live chat platforms as to naturally assess student production of speech 
acts. 

As with any study, certain limitations exist. Convincing students to complete two 30- 
minute live conversations with native speakers outside of the classroom proved to be the largest 
challenge in carrying out the study. While TalkAbroad was a required component of all Spanish 


courses at School X, there is a noticeable trend of students not completing one or both of these 
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assignments, perhaps due to the cost, time, or the anxiety involved. As such, data from a 
significant amount of otherwise consenting participants were not used due to their failure to 
complete all portions of the study. Further, while most students reverted to the Anglo-Saxon 
‘thank you,’ a number of students did not acknowledge the compliment and chose not to 
respond. This was likely due to their inability to comprehend the compliment rather than a 
pragmatic strategy employed. Additional limitations include the length of the study (one 
semester), the idea of “standard Spanish’ or a ‘correct’ pragmatic response, and the nature of the 
MCDCT. 

The researcher recognises that, while the MCDCT has demonstrated to be a reliable tool 
for assessing pragmatic speech acts in prior studies (Bachelor, 2016; Bachelor, 2015; Liu, 2007; 
Jianda, 2006; Roever, 2006; Roever, 2005), pragmatic recognition is assessed rather than 
production. 

Additionally, the grading criteria for the TalkAbroad conversations (see Research 
questions and data analysis procedures) and the idea of a ‘correct’ response on the MCDCT rely 
on the idea of a ‘standard Spanish.’ While there is no consensus regarding what is considered 
pragmatically appropriate, the researcher relies on the studies by Smith (2009), Maiz-Arévalo 
(2010), Lorenzo-Dus (2001), and Ramajo (2012) and on the responses provided by native 
speakers during the MCDCT pilot to reach his conclusions on a ‘correct’ response. Future 
investigations that address these limitations should most definitely be carried out in the near 
future. 

As stated by Van Houten and Shelton (2018, p. 38), the journey to Intercultural 
Communicative Competence is ‘a personal one, with many steps, both backward and forward, 


and a growing awareness of self and other. Just as the use of the language “Can-Do Statements” 
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has had a positive impact on learning and teaching, the “NCSSFL-ACTFL Can-Do Statements 
for Intercultural Communication” are expected to make both learner and educator more mindful 


of the importance of culture in communication.’ 
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Appendix A 


Multiple-Choice Discourse Completion Task (MCDCT) 

Read each scenario and choose the most culturally appropriate response by selecting A, B, or C. 
This does not count towards your grade and is for research-purposes only, so please do not use 
handouts, the textbook, dictionaries, friends, translators, etc. 


1) Your best friend has a new haircut (corte de pelo) that you really like. 
You say: 

A. Me gusta. 

B. Te queda muy bien el corte de pelo. 

C. jBuen corte de pelo! 


2) You bought new shoes and your sister tells you she really likes them. 
You say: 

A. Gracias. Si quieres te los presto. 

B. Gracias. 

C. [Say nothing and smile] 


3) Your mom made dinner using a new recipe and you really like it. 
You say: 

A. jQué rico! ;Te ha quedado super bien! 

B. Me gusta mucho. 

C. La cena esta muy rica; me gusta. 


4) You are going to tutoring for Spanish and your tutor tells you that he’s really impressed with 
your progress. 
You say: 

A. Muchas gracias. 

B. Bueno, es mérito del tutor. 

C. ~A que si? 


5) Your English professor gave a speech (discurso) to the campus yesterday and you were really 
impressed and want to tell her. 
You say: 

A. Me gust6 mucho. 

B. jQué buen discurso! 

C. Me encanto su discurso. Fue muy motivador. 


6) You made cookies for your Spanish class and everyone is telling you that they’re the best 
cookies they’ve ever tried. 
You say: 

A. [Say nothing and smile] 

B. Muchas gracias. 

C. Gracias; no tiene nada, la verdad; son muy faciles de hacer. 
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7) Your dad just bought a new car (coche) and you love it. 
You say: 
A. {Qué coche mas bonito! ;Me lo tienes que prestar! 
B. ;Me encanta tu coche! 
C. Es un coche muy bonito. 


8) You run into the college president who tells you that he’s really impressed with an article that 
you wrote for the school newspaper. 
You say: 

A. [Say nothing and smile] 

B. Muchas gracias; es un gusto poder participar en el periddico. 

C. Muchas gracias. 


9) One of your coworkers has been really exceeding at work lately and you want to tell her. 
You say: 

A. Buen trabajo. 

B. Te felicito por el buen trabajo que has estado haciendo. 

C. Trabajas bien. 


10) You bought new furniture for your apartment and your landlord tells you that you should be 
a designer. 
You say: 

A. Gracias. 

B. [Say nothing and smile] 

C. ¢Td crees? Encontré las ideas en una revista de disefio. 
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Appendix B 


Lesson 1 — Compliments & Compliment Responses 
What is a compliment? 
Compliments are common forms of communication in both English and Spanish. However, there 
are slight differences in how each language realizes these actions. A compliment is when the 
speaker says something nice to the hearer about his/her appearance, abilities, among other things. 


Use of compliments 

In Spanish, compliments are frequently carried out using a limited number of adjectives and 
verbs and can be categorized by those on abilities (i.e., you speak Spanish really well) and by 
those on appearance. 


The most frequent adjectives used to compliment in Spanish are “bueno/a,’ ‘bonito/a,’ ‘guapo/a,’ 
‘rico/a,’ and ‘inteligente.’ In terms of speech patterns, Spanish-speakers will often use ‘qué’ plus 
an adjective, as in ‘qué bonito,’ among other structures. While English speakers frequently say, 
“you look nice’ or ‘your hair looks great,’ Spanish speakers prefer the ‘qué’ with an adjective 
structure. Thus, ‘you look so pretty’ would likely be translated as ‘;qué guapa!’ 


Complimenting in Spanish 


;Qué guapa! > Qué + Adjective 

{Qué bonito carro! > Qué + Adjective + Noun Phrase 

{Qué bonito se te ve el pelo! > Qué + Adjective + Verb + Noun Phrase 
jTienes unos ojos muy bonitos! > Verb + Noun Phrase + (Adverb) Adjective 
jEsta sopa te salié rica! > Noun Phrase + Verb + Adjective 

iTu trabajo estuvo muy bien > Noun Phrase + Verb + Adverb 


ON NS 


In terms of compliment responses, most studies have concluded that Spanish speakers are much 
more likely than speakers of English to utilize strategies like refusing, downgrading, and 
questioning in responding to compliments. 


For example, let’s say you made brownies for your workmates and one of them tells you ‘jQué 
brownies mas deliciosos! ;Te salieron muy ricos!’ [What delicious brownies! They turned out 
really good!|. An English speaker would likely accept the compliment and say something like, 
‘Thanks, glad you like them!’ 


While it’s not wrong to accept a compliment in Spanish, a Spanish speaker would be much more 
likely to downgrade the compliment: ‘Bueno; son muy faciles de hacer’ [Well, they’re really 
easy to make]. In this example, the hearer acknowledged the compliment (Bueno), but made it 
seem like it was no big deal (they were really easy to make), thus downgrading it. 
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Some common compliment response strategies in Spanish 
Strategies Sub strategies 
Acceptance Showing agreement 
Giving an explanation or information 


Mitigation Downgrading 
Doubting 
Questioning 
Asking for reaffirmation 


Exercise 1 - Structures 

Translate the following compliments: 
Example: You look pretty! --> 

(Qué bonita! 


A) You look handsome! --> 

B) (talking about cookies) These are so good! --> 

C) Your hair looks great --> 

D) (talking about a friend’s shirt) It looks really nice --> 


Exerise 2 - Practice 

Complete the following with an appropriate form of these formulas to respond to a compliment. 
1) A classmate tells you that the cookies you made taste awesome. You reply: 

Bueno.. 

2) Your friend compliments ms on oli new outfit. You reply: 

Ah, jestas... 
3) A workmate says he’s Sani you’ve improved a lot at work lately. You reply: 
iT crees? 

4) Your brother compliments you on ayouk new shoes. You reply: 


Exercise 3 - In context 
Read the following conversation and answer the questions afterwards. 


Jorge y Roberto son estudiantes en la Universidad Complutense de Madrid. Jorge tiene unos 
zapatos nuevos y a Roberto le gustan. 


Jorge: Hola Roberto, {como estas? 

Roberto: Estoy bien, zy tt? 

Jorge: Bien, {qué tal en la universidad? 

Roberto: Ya sabes, lo de siempre... (Qué te iba a decir? jQué zapatos mas guapos! 
Jorge: {TU crees? 

Roberto: jClaro que si! El color, el estilo, todo, todo... 

Jorge: Bueno, son de una tienda pequefia cerca de mi casa. 

Roberto: En serio, te quedan muy bien. 
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1) What purpose does Roberto have in this conversation? 

2) What key words led you to categorize the conversation as you did? 

3) What roles did each of the speakers have? 

4) Did Jorge ever directly accept the compliment? What response strategies did he use? 


5) Was the conversation successful? Was the goal reached? 


36 


Appendix C 
Compliments & Compliment Responses in Spanish: Lesson 2 


Read the following two scenarios in which a friend compliments another friend on her shoes. 
Scenario 1: 

Friend 1: ;Qué zapatos mas bonitos! 

Friend 2: {Td crees? 

Friend 1: Es que a ti te queda bien todo. 

Friend 2: Bueno, ja ti también te queda todo bien! 

Scenario 2: 

Friend 1: Me gustan tus zapatos. 

Friend 2: Ay, gracias. 

Friend 1: De nada 


Which conversation do you perceive as more effective? Having in mind that Spanish speakers 
often compliment each other to break the ice in a conversation, were you right about which 
scenario was likely more effective? In Spanish, speakers frequently use “qué + adjective’ to 
compliment others. Often, the recipient of the compliment utilizes strategies to downplay the 
compliment, doesn’t directly accept the compliment, or returns the compliment. What were the 
two compliments that were given? What were the two responses and how would you classify 
them? 


Now, we are going to watch 3 YouTube clips from Spanish-speaking countries where 
compliments are given. 
1. Clip from ‘Yo Soy,’ a Peruvian show where contestants try to win a prize by imitating 


famous people. In this clip, the contestant lets one of the judges know that he’s been 
watching the show just to see her and can’t believe she’s in front of him. Watch the clip 
(1:59-2:15, https://www.youtube.com/watch?v=_ hAIvM_emD0) 


Script: 

Contestant: Me dejas en blanco. Yo estaba viendo desde Espafia este programa todos los 
dias por verte ati y ahora que te tengo en frente no me lo creo todavia 

Judge: Qué lindo eres pero ya no creo en esas palabras 


What is the relationship between the speakers? Are they friends? What was the point of 
the compliment? How did the judge respond? What strategy did she use? Was the 
contestant successful in his compliment? 


2. Clip froma YouTube video where speakers walk up to their classmates by surprise, tell 
them that they have a question, and then give them a compliment to see how they react. 
See how this person reacts when he is told that he looks nicer than usual today (2:44- 
3:08, https://www.youtube.com/watch?v=dJur_Sjtr8g) 
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Script: 

Pl: (Qué te hiciste hoy, que estas mas lindo? 

P2: {Sera? No sé, pues, tal vez... No, pues, creo que es lo normal, como uno se viste, a 
veces un poco... Pues, no sé, depende de la ocasidn. Digamos me toc6 madrugar. Hoy no 
me siento que... hoy siento que no me he alistado tanto, o sea que no me ... 

P1: Pero estas mas lindo, pero estas mas lindo 

P3: Si, yo estoy de acuerdo 

P2: (Esa es la pregunta? 

Pl: Si 

P2: Bueno 


What is the relationship between the speakers? Are they friends? How did person 2 
respond? What strategies did he use? Did person 2 ever directly accept the compliment? 


Clip 3 from the Spanish TV show, ‘Aqui no hay quien viva.’ Three people return home 
after shopping and compliments are given about the other person’s clothes and 
appearance (44:46-45:00, https://www.youtube.com/watch?v=QvE9O Xgk7KE) 


P1: Los pantalones negros no me gustaban nada. 

P2: No, a mi tampoco 

P1: Pero esa camiseta que te has comprado te queda fenomenal 

P2: (De verdad? 

P1: Es que Yolanda tiene un cuerpazo, {a q si, Bea? 

3: (Eh? Si si, un cuerpazo 

P2: Bueno, tt también tienes un tipo. Si quieres te puedo prestar algo de ropa. {Qué talla 
usas? 


What is the relationship between the speakers? Are they friends? What was the point of 
the compliment? How did P2 respond? What strategy did she use? 
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Tables 
Table 1. 
Complimenting in Spanish (Félix-Brasdefer, 2011, para 6). 
1. j;Qué guapa! [How pretty!] > Qué + Adjective 


. {Qué bonito carro! [What a nice car!] > Qué + Adjective + Noun Phrase 
3. jQué bonito se te ve el pelo! [Your hair looks nice!] > Qué + Adjective + Verb + Noun 


Phrase 

4. Tienes unos ojos muy bonitos! [You have very pretty eyes!] > Verb + Noun Phrase + 
(Adverb) Adjective 

5. jEsta sopa te salié rica! [Your soup turned out delicious!] > Noun Phrase + Verb + 
Adjective 

6. jTu trabajo estuvo muy bien! [Your work was really good!] > Noun Phrase + Verb + 
Adverb 

Table 2. 


Common compliment response strategies in Spanish 


Macro-strategies Sub strategies 
Acceptance Showing agreement 
Giving an explanation or information 


Mitigation Downgrading 
Doubting 
Questioning 
Asking for reaffirmation 


Table 3. 
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Student responses to compliments during post-intervention TalkAbroad conversations 


Student Group Response 

Student 1 1 (control) Si, me encanta aprender espajiol 

Student 2 1 (control) Si 

Student 3 1 (control) Gracias, gracias 

Student 4 1 (control) Gracias 

Student 5 1 (control) No response 

Student 6 1 (control) Gracias 

Student 7 1 (control) No response 

Student 8 1 (control) Si 

Student 9 2 (experimental) No response 

Student 10 2 (experimental) No response 

Student 11 2 (experimental) Gracias 

Student 12 2 (experimental) Gracias / si 

Student 13 2 (experimental) No response 

Student 14 2 (experimental) Gracias 

Student 15 2 (experimental) Oh, gracias 

Student 16 2 (experimental) Ok 

Student 17 2 (experimental) Ah, no ... no soy bueno en espafiol 

Student 18 2 (experimental) No response 

Student 19 2 (experimental) Gracias 

Student 20 2 (experimental) Gracias 

Student 21 2 (experimental) Gracias 

Student 22 2 (experimental) Si, gracias por...si, si 

Student 23 2 (experimental) Gracias, igualmente 

Student 24 3 (intermediate/experimental) | Si 

Student 25 3 (intermediate/experimental) | Gracias 

Student 26 3 (intermediate/experimental) | Ah, mas o menos. Estudié espafiol 
para cuatro afios en secundario pero 
en la universidad es muy dificil. Y 
mis profesoras diferentes cada afio y 
la transiciOn es muy dificil para mi 

Student 27 3 (intermediate/experimental) | Gracias 

Student 28 3 (intermediate/experimental) _| Muchas gracias 

Student 29 3 (intermediate/experimental) | Gracias 

Student 30 3 (intermediate/experimental) | Gracias 


